Search Results: "robin"

7 September 2017

Reproducible builds folks: Reproducible Builds: Weekly report #123

Here's what happened in the Reproducible Builds effort between Sunday August 27 and Saturday September 2 2017: Talks and presentations Holger Levsen talked about our progress and our still-far goals at BornHack 2017 (Video). Toolchain development and fixes The Debian FTP archive will now reject changelogs where different entries have the same timestamps. UDD now uses reproducible-tracker.json (~25MB) which ignores our tests for Debian unstable, instead of our full set of results in reproducible.json. Our tests for Debian unstable uses a stricter definition of "reproducible" than what was recently added to Debian policy, and these stricter tests are currently more unreliable. Packages reviewed and fixed, and bugs filed Patches sent upstream: Debian bugs filed: Debian packages NMU-uploaded: Reviews of unreproducible packages 25 package reviews have been added, 50 have been updated and 86 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by: diffoscope development Version 86 was uploaded to unstable by Mattia Rizzolo. It included previous weeks' contributions from: reprotest development Development continued in git with contributions from: Misc. This week's edition was written by Ximin Luo, Chris Lamb, Bernhard M. Wiedemann and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

9 July 2017

Steinar H. Gunderson: Nageru 1.6.1 released

I've released version 1.6.1 of Nageru, my live video mixer. Now that Solskogen is coming up, there's been a lot of activity on the Nageru front, but hopefully everything is actually coming together now. Testing has been good, but we'll see whether it stands up to the battle-hardening of the real world or not. Hopefully I won't be needing any last-minute patches. :-) Besides the previously promised Prometheus metrics (1.6.1 ships with a rather extensive set, as well as an example Grafana dashboard) and frame queue management improvements, a surprising late addition was that of a new transcoder called Kaeru (following the naming style of Nageru itself, from the japanese verb kaeru ( ) which means roughly to replace or excahnge iKnow! claims it can also mean convert , but I haven't seen support for this anywhere else). Normally, when I do streams, I just let Nageru do its thing and send out a single 720p60 stream (occasionally 1080p), usually around 5 Mbit/sec; less than that doesn't really give good enough quality for the high-movement scenarios I'm after. But Solskogen is different in that there's a pretty diverse audience when it comes to networking conditions; even though I have a few mirrors spread around the world (and some JavaScript to automatically pick the fastest one; DNS round-robin is really quite useless here!), not all viewers can sustain such a bitrate. Thus, there's also a 480p variant with around 1 Mbit/sec or so, and it needs to come from somewhere. Traditionally, I've been using VLC for this, but streaming is really a niche thing for VLC. I've been told it will be an increased focus for 4.0 now that 3.0 is getting out the door, but over the last few years, there's been a constant trickle of little issues that have been breaking my transcoding pipeline. My solution for this was to simply never update VLC, but now that I'm up to stretch, this didn't really work anymore, and I'd been toying around with the idea of making a standalone transcoder for a while. (You'd ask why not the ffmpeg(1) command-line client? , but it's a bit too centered around files and not streams; I use it for converting to HLS for iOS devices, but it has a nasty habit of I/O blocking real work, and its HTTP server really isn't meant for production work. I could survive the latter if it supported Metacube and I could feed it into Cubemap, but it doesn't.) It turned out Nageru had already grown most of the pieces I needed; it had video decoding through FFmpeg, x264 encoding with speed control (so that it automatically picks the best preset the machine can sustain at any given time) and muxing, audio encoding, proper threading everywhere, and a usable HTTP server that could output Metacube. All that was required was to add audio decoding to the FFmpeg input, and then replace the GPU-based mixer and GUI with a very simple driver that just connects the decoders to the encoders. (This means it runs fine on a headless server with no GPU, but it also means you'll get FFmpeg's scaling, which isn't as pretty or fast as Nageru's. I think it's an okay tradeoff.) All in all, this was only about 250 lines of delta, which pales compared to the ~28000 lines of delta that are between 1.3.1 (used for last Solskogen) and 1.6.1. It only supports a rather limited set of Prometheus metrics, and it has some limitations, but it seems to be stable and deliver pretty good quality. I've denoted it experimental for now, but overall, I'm quite happy with how it turned out, and I'll be using it for Solskogen. Nageru 1.6.1 is on its way into Debian, but it depends on a new version of Movit which needs to go through the NEW queue (a soname bump), so it might be a few days. In the meantime, I'll be busy preparing for Solskogen. :-)

4 May 2017

Guido G nther: Debian Fun in April 2017

Debian LTS April marked the 24th month I contributed to Debian LTS under the Freexian umbrella. I had 8 hours allocated plus 4 hours left from March which I used by: Other Debian stuff git-buildpackage Released versions 0.8.14 and 0.8.15. Notable changes besides bug fixes: The versions are also available on pypi.

2 May 2017

Russ Allbery: Review: Ninefox Gambit

Review: Ninefox Gambit, by Yoon Ha Lee
Series: The Machineries of Empire #1
Publisher: Solaris
Copyright: 2016
ISBN: 1-84997-992-8
Format: Kindle
Pages: 384
Charis is a Kel, which means that she's a soldier of the Hexarchate. A captain, to be precise: Captain Kel Charis of Heron Company, commanding infantry forces to stamp out heresy. The heresy she's stamping out at the start of the book are the Eels, or the Society of the Flourish as they call themselves, and they're strong enough that they can also command heretical technology. Charis manages to win through, but only because she has enough flexibility and quick thinking to adjust to the presence of a heresy and reach beyond the Lexicon Primary, improvising formations on the spot to adjust for the affects of the rebel calendar. Military victory is prized among the Kel, but stepping outside the bounds of Doctrine to achieve it is not. Charis is not particularly surprised when her company is disbanded for re-education after the battle. She is very surprised when she personally is tapped to offer a solution to a far greater attack on the Hexarchate. I first encountered Yoon Ha Lee's fiction in the short story "The Unstrung Zither", was blown away by the creativity and delightfully weird twist on science fiction war, and have been following his writing ever since. Most of it is short fiction, though, and I'm not much of a short fiction reader, so there haven't been many reviews. Ninefox Gambit is his first, and much-anticipated, full-length novel. It's probably not too surprising for someone from the generation that grew up with Star Wars, but I have a soft spot in my heart for magitech. Hard science fiction has its merits, as does the softer sort that takes standard, if impossible, genre tropes for granted. But something about a far-future, space-faring society based on magic that straddles the rules of technology, physics, affinities, and beliefs calls to that part of me that spent hours thinking about the nature of the Force. It has to be good magitech, though: something odd and different but well-thought-out, full of implications and twisty consequences that reshape society and that inspire a whole new type of engineering and science. Magic that's not physics as we know it, but that's knowable, researchable, and something that a society can reshape itself around. This is the good magitech. In the world of the Hexarchate, calendrical systems are more than just a mutual agreement for conveying time. They order and structure the laws of the universe as much as they structure society. What technological devices, and what weapons, are possible is influenced by the calendar in observance, which in turn is based on what calendar people believe in and follow. Close adherence to a calendrical regime enables exotics: weapons with incredibly powerful and often horrific effects, such as the threshold winnower that plays a repeated, nightmarish role in this story. Invariant weapons, ones that will work in any calendar system, are much weaker. The Hexarchate is called that because it is a society formed by six factions, who divide the work of ruling its scattered planets according to the expertise and tendencies of each faction. Together, they impose the high calendar, and maintain it against heresies with an iron fist lest their power be undermined or transformed and their exotics fail to function. The Kel is their military faction, a key component of that fist, and their specialty is formations: specific arrangements of humans or ships that channel the power of the calendar to defend against or attack with exotics. Formations have to be held exactly to hold their power and yet have to be flexible enough to change based on fast-changing battlefield conditions. To assist in this, Kel are programmed with formation instinct: psychological conditioning that helps them obediently take and hold formations. And, not coincidentally, offer nearly absolute obedience to their chain of command. I just finished reading another book that attempted to use math as a key component of its world-building. I think Lee was far more successful. The math here is realistic for its purpose, obviously necessary given the formation structures built into the world's physics, and a lovely nod to the importance of calendars. A single calendar might involve only simple arithmetic, but the formation and technological implications of a calendar, let alone the fuzzy boundaries between two calendars each partially in force, would naturally require tricky advanced mathematics to work out. For someone in Charis's position, mathematical training is a rare but vitally important tactical advantage. As you might have guessed from the amount that I'm talking about combat, Ninefox Gambit is military SF. Charis is a military officer, and a comfortable majority of this book is combat of one kind or another. That's not normally my thing, and I did wish there was a bit more non-military social development. But my normal problem with military SF is that I lack the interest in battlefield tactics and strategy to stay fully engaged by description of battle after battle. Ninefox Gambit is the story of Charis attempting to retake a stronghold of the Hexarchate that's fallen to heretical forces, but Lee adds an important twist that does keep me engaged: Jedao. General Shuos Jedao was the greatest general the Hexarchate had ever seen. He never lost a battle. The only catch is that, in the middle of a highly successful campaign against heretics, he went mad, slaughtering both the heretics and his own troops with a horrific weapon while simultaneously murdering all of his command staff. He's much too dangerous and insane to leave alive, but he was also much too valuable and skillful to lose as a weapon, so for the subsequent centuries he's been kept in a threshold state, an undead ghost. A ghost that the Hexarchate can put into Charis's head, a constant advisor as she's placed in charge of the swarm sent to retake the Fortress of Scattered Needles. A brilliant tactician, sociopath, and mass murderer whose advice can never be trusted. The heart of Ninefox Gambit isn't the combat. It's the interplay of power, analysis, and guesswork under the combat, as Charis attempts to use Jedao's brilliance while not losing her own sense of identity or letting him mess too badly with her head. At the start, she's way out of her depth. But she's thoughtful, careful, has a strong internal sense of identity, and learns fast. And the story of Jedao's past is accurate, but incomplete. For those who are familiar with the often-ornate language and prose style of Yoon Ha Lee's short fiction and who are worried it wouldn't hold up at longer length, note that his style here is much different. There are a few touches of ornate description, but most of the book is written in a straightforward and easily-understandable narrative style. Thankfully, because the layers of tactical thrust and counter-thrust are complicated enough that I would have lost them entirely beneath too-complex prose. There's a lot of brutal death in this book. I got a bit tired of both that and the tactical maneuvering, although that's less the fault of the book and more my own mild antipathy towards military SF. But the unique universe background held my interest long enough to become intrigued by Charis's slow but determined probing at Jedao's secrets and the politics of the Hexarchate. I still would have preferred the story to have a somewhat lower body count, but as long as one can read past some gore, there's plenty here to appeal to someone who normally gives military SF a pass. I think its biggest drawback is that, although it has a narrative arc that comes to a clear conclusion, Ninefox Gambit raises a lot of important questions about its world and mostly doesn't answer them. There are more books coming, and I hope they contain more definitive answers. Followed by Raven Stratagem. Rating: 7 out of 10

1 May 2017

Russ Allbery: Review: Survival

Review: Survival, by Julie E. Czerneda
Series: Species Imperative #1
Publisher: DAW
Copyright: May 2004
ISBN: 0-7564-0180-1
Format: Hardcover
Pages: 401
Dr. Mackenzie Connor, Mac to everyone she works with, is a biodiversity researcher specializing in salmon. In her future United States, humanity seems to have caught on to the importance of preserving wild places and learning about them, and is willing to invest in good equipment and a semi-permanent research installation. This comes with some occasional drawbacks, since she has to fight to get access to the salmon runs inside a nature preserve, but she wouldn't have it any other way. She wins enough of those fights, won the latest, and is now in position to monitor a run in a way that she's never been able to before. She was not expecting an alien to go diving in the middle of her salmon run. She was certainly not expecting that alien to be accompanied by a bureaucrat insisting that this alien's curiosity is more important than her research (hah). But the accompanying letter she receives is scarily persuasive, if maddeningly unhelpful. Much like the apparently jovial, earnest, eager, and very odd alien. Mac's continued hopes that she can quickly put this bizarre intrusion to rest and go back to her salmon are dashed by an impossible power outage, an alien visitor, and a violent kidnapping. Now her best friend and colleague is missing, the bureaucrat is not who he appears to be, and Mac is getting caught up in something that feels way over her head. SF novels feature a lot of science, but not a lot of scientific research. The research that does appear is often impulsive, wildly compressed, or far too focused on the breakthroughs of single people. The SF novel that everyone points to for accurate portrayal of real scientific research is Benford's Timescape, which I found deeply unexciting. Now I have a new novel to point to for a better treatment, although (somewhat disappointingly) Mac's research gets sidelined relatively early in the story and left behind for the conclusion. Czerneda gives us not just a few scientists and an imaginary research project, but an entire operational field station with a history. The Norcoast Salmon Research Facility is located just off-shore in carefully-designed domes to provide easy access to the sea with minimal intrusion into the local ecology. It's a bustling mix of research scientists, engineers, and the ever-present seasonal grad students, who come and go in all their immature enthusiasm and are viewed with a motherly bemusement that I immediately recognized from years of working at a university. Mac splits administrative duties with another scientist in an arrangement that will be familiar to academics everywhere, and the book opens with a mutually suspicious but mostly scripted turf fight with the guardian of the neighboring wildlife trust, the same fight they've been having every six months for years. I know Czerneda is herself a biologist by training; I'm not sure what her other academic background is, but if she hasn't spent years around academics and field studies, she's at least done some excellent research. A lot of novels have a quotidian background that's interrupted by the arrival of the plot. At the start of the story, the characters often care more about their day-to-day lives than the plot, and are dragged into it reluctantly. But one sign of an excellent writer is their ability to get the reader to care about that quotidian background alongside the character, and to sympathize with the character's reluctance to get pulled into the promised (and generally more exciting) novel plot. Czerneda succeeds in this about as well as any writer I've read since Robin McKinley's Sunshine, and that's high praise. I cared about Mac's salmon, I was nearly as irritated as she was when her research was interrupted, and I still want to go back and see more of the experiments and studies she was hoping to run. Interstellar drama and threats to multiple species are all well and good, but the salmon are running! The actual plot is a mysterious threat that turns into a combination of a biological and cultural puzzle and a sort of first-contact story. Mac is not truly the first human to encounter the Dhryn, but she's certainly the first person they've explained anything to, and the first human to go where she goes. Sadly, it also shares some of the characteristics that sour me a bit on biological SF for personal reasons: a bit too much description of food, eating habits, squishy body parts, digestive processes, and biological discomfort. This is mostly a personal gripe, and won't bother other people as much as it does me, but I could have done without bits like the descriptions of Mac's attempts to figure out how to survive on alien cuisine. I'm also dubious of some of the biology of the Dhryn; given the startling bizarreness of Earth biology, maybe I shouldn't be, but I still think there are a few problems with the square-cube law here. But Mac's irrepressible grumpy curiosity makes this story, even in the bits that made me squeamish. I think I'd read any book in which she's the main character. I will warn that the ending is surprisingly dark and wasn't what I was expecting, and Survival doesn't resolve its central mysteries. This is clearly the first book of a trilogy and should be read with that expectation. But I thoroughly enjoyed it, and hopefully the next book will have more salmon. Followed by Migration. Rating: 8 out of 10

27 April 2017

Russ Allbery: Review: Necessity

Review: Necessity, by Jo Walton
Series: Thessaly #3
Publisher: Tor
Copyright: July 2016
ISBN: 0-7653-7902-3
Format: Hardcover
Pages: 331
Athena's experiment with a city (now civilization) modeled after Plato's Republic continues, but in a form that she would not have anticipated, and in a place rather far removed from its origins. But despite new awareness of the place and role of gods, a rather dramatic relocation, and unanticipated science-fiction complications, it continues in much the same style as in The Just City: thoughtful, questioning debate, a legal and social system that works surprisingly well, and a surprising lack of drama. At least, that is, until the displaced cities are contacted by the mainstream of humanity, and Athena goes unexpectedly missing. The latter event turns out to have much more to do with the story than the former, and I regret that. Analyzing mainline human civilization and negotiating the parameters of a very odd first contact would have, at least in my opinion, lined up a bit better with the strengths of this series. Instead, the focus is primarily on metaphysics, and the key climactic moment in those metaphysics is rather mushy and incoherent compared to the sharp-edged analysis Walton's civilization is normally capable of. Not particularly unexpected, as metaphysics of this sort are notoriously tricky to approach via dialectical logic, but it was a bit of a letdown. Much of this book deals with Athena's disappearance and its consequences (including the title), and it wasn't bad, but it wanders a bit into philosophical musings on the nature of gods. Necessity is a rather odd book, and I think anyone who started here would be baffled, but it does make a surprising amount of sense in the context of the series. Skipping ahead to here seems like a truly bad idea, but reading the entire series (relatively closely together) does show a coherent philosophical, moral, and social arc. The Just City opens with Apollo confronted by the idea of individual significance: what does it mean to treat other people as one's equals in an ethical sense, even if they aren't on measures of raw power? The Thessaly series holds to that theme throughout and follows its implications. Many of the bizarre things that happen in this series seem like matter-of-fact outcomes once you're engrossed in the premises and circumstances at the time. Necessity adds a surprising amount of more typical science fiction trappings, but they turn out to be ancillary to the story. What matters is considered action, trying to be your best self, and the earnest efforts of a society to put those principles first. And that's the strength of the whole series, including Necessity: I like these people, I like how they think, and I enjoy spending time with them, almost no matter what they're doing. As with the previous books, we get interwoven chapters from different viewpoints, this time from three primary characters plus some important "guest" chapters. As with the previous books, the viewpoint characters are different again, mostly a generation younger, and I had to overcome my initial disappointment at not hearing the same voices. But Walton is excellent at characterization. I really like this earnest, thoughtful, oddly-structured society that always teeters on the edge of being hopelessly naive and trusting but is self-aware enough to never fall in. By the end of the book, I liked this round of characters nearly as much as I liked the previous rounds (although I've still never liked a character in these books as well as I liked Simmea). I think one incomplete but important way to sum up the entire Thessaly series is that it's a trilogy of philosophical society-building on top of the premise of a universal love for and earnest, probing, thoughtful analysis of philosophy. Walton's initial cheat is to use an deus ex machina to jumpstart such a society from a complex human world that would be unlikely to provide enough time or space for it to build its own separate culture and tradition. I think the science-fiction trick is required to make this work real-world societies that try this end up having to spend so much of their energy fighting intrusion from the outside and diffusion into the surrounding culture that they don't have the same room to avoid conformity and test and argue against their own visions. Necessity is not at all the conclusion of that experiment I would expect, but it won me over, and I think it worked, even if a few bits of it felt indulgent. Most importantly for that overall project, this series is generational, and Necessity shows how it would feel to grow up deep inside it, seeing evolution on top of a base structure that is ubiquitous and ignored. Even the generation in The Philosopher Kings wasn't far enough removed to support that; Necessity is, and in a way this book shows how distinctly different and even alien human culture can become when it has space to evolve on top of different premises. I enjoyed the moments of small surprise, where characters didn't react the way that I'd expect for reasons now buried generations-deep in their philosophical foundations. This book will not win you over if you didn't already like the series, and I suspect it will lose a few people who read the previous two books. The plot structure is a little strange, the metaphysics are a touch strained, and the ending is, well, not quite the payoff that I was hoping for, although it's thematically appropriate and grew on me after a few days of thinking it over. But I got more Socrates, finally, who is as delightful as always and sorely needed to add some irreverence and contrariness to the the mix. And I got to read more about practical, thoughtful people who are trying hard to do their best, to be their best selves, and to analyze and understand the world. There's something calming, delightful, and beautifully optimistic about their approach, and I'm rather sad to not have more of it to read. Rating: 7 out of 10

17 April 2017

Russell Coker: More KVM Modules Configuration

Last year I blogged about blacklisting a video driver so that KVM virtual machines didn t go into graphics mode [1]. Now I ve been working on some other things to make virtual machines run better. I use the same initramfs for the physical hardware as for the virtual machines. So I need to remove modules that are needed for booting the physical hardware from the VMs as well as other modules that get dragged in by systemd and other things. One significant saving from this is that I use BTRFS for the physical machine and the BTRFS driver takes 1M of RAM! The first thing I did to reduce the number of modules was to edit /etc/initramfs-tools/initramfs.conf and change MODULES=most to MODULES=dep . This significantly reduced the number of modules loaded and also stopped the initramfs from probing for a non-existant floppy drive which added about 20 seconds to the boot. Note that this will result in your initramfs not supporting different hardware. So if you plan to take a hard drive out of your desktop PC and install it in another PC this could be bad for you, but for servers it s OK as that sort of upgrade is uncommon for servers and only done with some planning (such as creating an initramfs just for the migration). I put the following rmmod commands in /etc/rc.local to remove modules that are automatically loaded:
rmmod btrfs
rmmod evdev
rmmod lrw
rmmod glue_helper
rmmod ablk_helper
rmmod aes_x86_64
rmmod ecb
rmmod xor
rmmod raid6_pq
rmmod cryptd
rmmod gf128mul
rmmod ata_generic
rmmod ata_piix
rmmod i2c_piix4
rmmod libata
rmmod scsi_mod In /etc/modprobe.d/blacklist.conf I have the following lines to stop drivers being loaded. The first line is to stop the video mode being set and the rest are just to save space. One thing that inspired me to do this is that the parallel port driver gave a kernel error when it loaded and tried to access non-existant hardware.
blacklist bochs_drm
blacklist joydev
blacklist ppdev
blacklist sg
blacklist psmouse
blacklist pcspkr
blacklist sr_mod
blacklist acpi_cpufreq
blacklist cdrom
blacklist tpm
blacklist tpm_tis
blacklist floppy
blacklist parport_pc
blacklist serio_raw
blacklist button On the physical machine I have the following in /etc/modprobe.d/blacklist.conf. Most of this is to prevent loading of filesystem drivers when making an initramfs. I do this because I know there s never going to be any need for CDs, parallel devices, graphics, or strange block devices in a server room. I wouldn t do any of this for a desktop workstation or laptop.
blacklist ppdev
blacklist parport_pc
blacklist cdrom
blacklist sr_mod
blacklist nouveau blacklist ufs
blacklist qnx4
blacklist hfsplus
blacklist hfs
blacklist minix
blacklist ntfs
blacklist jfs
blacklist xfs
Related posts:
  1. Video Mode and KVM I recently changed my KVM servers to use the kernel...
  2. Testing STONITH One problem that I have had in configuring Heartbeat clusters...
  3. Modules and NFS for Xen I m just in the process of converting a multi-user system...

25 March 2017

Russ Allbery: Spring haul

Work has been hellishly busy lately, so that's pretty much all I've been doing. The major project I'm working on should be basically done in the next couple of weeks, though (fingers crossed), so maybe I'll be able to surface a bit more after that. In the meantime, I'm still acquiring books I don't have time to read, since that's my life. In this case, two great Humble Book Bundles were too good of a bargain to pass up. There are a bunch of books in here that I already own in paperback (and hence showed up in previous haul posts), but I'm running low on shelf room, so some of those paper copies may go to the used bookstore to make more space. Kelley Armstrong Lost Souls (sff)
Clive Barker Tortured Souls (horror)
Jim Butcher Working for Bigfoot (sff collection)
Octavia E. Butler Parable of the Sower (sff)
Octavia E. Butler Parable of the Talents (sff)
Octavia E. Butler Unexpected Stories (sff collection)
Octavia E. Butler Wild Seed (sff)
Jacqueline Carey One Hundred Ablutions (sff)
Richard Chizmar A Long December (sff collection)
Jo Clayton Skeen's Leap (sff)
Kate Elliot Jaran (sff)
Harlan Ellison Can & Can'tankerous (sff collection)
Diana Pharoh Francis Path of Fate (sff)
Mira Grant Final Girls (sff)
Elizabeth Hand Black Light (sff)
Elizabeth Hand Saffron & Brimstone (sff collection)
Elizabeth Hand Wylding Hall (sff)
Kevin Hearne The Purloined Poodle (sff)
Nalo Hopkinson Skin Folk (sff)
Katherine Kurtz Camber of Culdi (sff)
Katherine Kurtz Lammas Night (sff)
Joe R. Lansdale Fender Lizards (mainstream)
Robert McCammon The Border (sff)
Robin McKinley Beauty (sff)
Robin McKinley The Hero and the Crown (sff)
Robin McKinley Sunshine (sff)
Tim Powers Down and Out in Purgatory (sff)
Cherie Priest Jacaranda (sff)
Alastair Reynolds Deep Navigation (sff collection)
Pamela Sargent The Shore of Women (sff)
John Scalzi Miniatures (sff collection)
Lewis Shiner Glimpses (sff)
Angie Thomas The Hate U Give (mainstream)
Catherynne M. Valente The Bread We Eat in Dreams (sff collection)
Connie Willis The Winds of Marble Arch (sff collection)
M.K. Wren Sword of the Lamb (sff)
M.K. Wren Shadow of the Swan (sff)
M.K. Wren House of the Wolf (sff)
Jane Yolen Sister Light, Sister Dark (sff)

5 March 2017

Shirish Agarwal: To say or not to say

Voltaire For people who are visually differently-abled, the above reads To learn who rules over you, simply find out who you are not allowed to criticize Voltaire wrote this either in late 16th century or early 17th century and those words were as apt in those times, as it is in these turbulent times as well. Update 05/03 According to @bla these words are attributable to a neo-nazi and apparently a child abuser. While I don t know the context in which it was shared, it describes the environment in which we are perfectly. Please see his comment for a link and better understanding. The below topic requires a bit of maturity, so if you are easily offended, feel free not to read further. While this week-end I was supposed to share about the recent Science Day celebrations that we did last week Science Day celebrations at GMRT Would explore it probably next week. This week the attempt is to share thoughts which had been simmering at the back of my mind for more than 2 weeks or more and whose answers are not clear to me. My buttons were pressed when Martin f. Kraft shared about a CoC violation and the steps taken therein. While it is easy to say with 20:20 hind-sight to say that the gentleman acted foolishly, I don t really know the circumstances to pass the judgement so quickly. In reality, while I didn t understand the joke in itself, I have to share some background by way of anecdotes as to why it isn t so easy for me to give a judgement call. a. I don t know the topics chosen by stand-up comedians in other countries, in India, most of the stand-up acts are either about dating or sex or somewhere in-between, which is lovingly given the name Leela (dance of life) in Indian mythology. I have been to several such acts over the years at different events, different occasions and 99.99% of the time I would see them dealing with pedophilia, necrophilia and all sorts of deviants in sexuality and people laughing wildly, but couple of times when the comedian shared the term sex with people, educated, probably more than a few world-travelled middle to higher-middle class people were shocked into silence. I had seen this not in once but 2-3 times in different environments and was left wondering just couple of years back Is sex such a bad word that people get easily shocked ? Then how is it that we have 1.25 billion + people in India. There had to be some people having sex. I don t think that all 1.25 billion people are test-tube babies. b. This actually was what lead to my quandary last year when my sharing of My Experience with Debian which I had carefully prepared for newbies, seeing seasoned debian people, I knew my lame observations wouldn t cut ice with them and hence had to share my actual story which involved a bit of porn. I was in two minds whether or not to say it till my eyes caught a t-shirt on which it was said We make porn or something to that effect. That helped me share my point. c. Which brings me to another point, it seems it is becoming increasingly difficult to talk about anything either before apologizing to everyone and not really knowing who will take offence at what and what the repercussions might be. In local sharings, I always start with a blanket apology that if I say something that offends you, please let me know afterwards so I can work on it. As the term goes You can t please everyone and that is what happens. Somebody sooner or later would take offence at something and re-interpret it in ways which I had not thought of. Charlie Chaplin - King of self-deprecating humor From the little sharings and interactions I have been part of, I find people take offence at the most innocuous things. For instance, one of the easy routes of not offending anyone is to use self-deprecating humour (or so I thought) either of my race, caste, class or even my issues with weight and each of the above would offend somebody. Charlie Chaplin didn t have those problems. If somebody is from my caste, I m portraying the caste in a certain light, a certain slant. If I m talking about weight issues, then anybody who is like me (fat) feels that the world is laughing at them rather than at me or they will be discriminated against. While I find the last point a bit valid, it leaves with me no tools and no humour. I neither have the observational powers or the skills that Kapil Sharma has and have to be me. While I have no clue what to do next, I feel the need to also share why humour is important in any sharing.- a. Break When any speaker uses humour, the idea is to take a break from a serious topic. It helps to break the monotony of the talk especially if the topic is full of jargon talk and new concepts. A small comedic relief brings the attendees attention back to the topic as it tends to wander in a long monotonous talk. b. Bridge Some of the better speakers use one or more humourous anecdote to explain and/or bridge the chasm between two different concepts. Some are able to produce humour on the fly while others like me have to rely on tried and tested methods. There is one another thing as well, humour is seems to be a mixture of social, cultural and political context and its very easy to have it back-fired upon you. For instance, I attempted humour on refugees, probably not the best topic to try humour in the current political climate, and predictably, it didn t go down well. I had to share and explain about Robin Williams slightly dark yet humorous tale in Moscow on the Hudson The film provides comedy and pathos in equal measure. You are left identifying with Vladimir Ivanoff (Robin Williams character) especially in the last scene where he learns of his grand-mother dying and he remembers her and his motherland, Russia and plays a piece on his saxophone as a tribute both to his grand-mother and the motherland. Apparently, in the height of the cold war, if a Russian defected to United States (land of Satan and other such terms used) you couldn t return to Russia. The movie, seen some years back left a deep impact on me. For all the shortcomings and ills that India has, even if I could, would and could I be happy anywhere else ? The answers are not so easy. With most NRI s (Non-Resident Indians) who emigrated for good did it not so much for themselves but for their children. So the children would hopefully have a better upbringing, better facilities, better opportunities than they would have got here. I talked to more than a few NRI s and while most of them give standardized answers, talking awhile and couple of beers or their favourite alcohol later, you come across deeply conflicted human beings whose heart is in India and their job, profession and money interests compel them to be in the country where they are serving. And Indian movies further don t make it easy for the Indian populace when trying to integrate into a new place. Some of the biggest hits of yesteryear s were about having the distinct Indian culture in their new country while the message of most countries is integration. I know of friends who are living in Germany who have to struggle through their German in order to be counted as a citizen, the same I guess is true of other countries as well, not just the language but the customs as well. They also probably struggle with learning more than one language and having an amalgamation of values which somehow they and their children have to make sense of. I was mildly shocked last week to learn that Mishi Choudary had to train people in the U.S. to differentiate between Afghan turban styles of wearing and the Punjabi style of wearing the turban. A simple search on Afghani turban and Punjabi turban reveals that there are a lot of differences between the two cultures. In fact, the way they talk, the way they walk, there are lots that differentiate the two cultures. The second shocking video was of an African-American man racially abusing an Indian-American girl. At first, I didn t believe it till I saw the video on facebook. My point through all that is it seems humour, that clean, simple exercise which brings a smile to you and uplifts the spirit doesn t seem to be as easy as it once was. Comments, suggestions, criticisms all are welcome.
Filed under: Miscellenous Tagged: #Elusive, #Fear, #hind-sight, #Humour, #immigrant, #integration, #Mishi Choudary, #refugee, #Robin Williams, #self-deprecating, #SFLC, #two-minds

27 December 2016

Joey Hess: multi-terminal applications

While learning about and configuring weechat this evening, I noticed a lot of complexity and unsatisfying tradeoffs related to its UI, its mouse support, and its built-in window system. Got to wondering what I'd do differently, if I wrote my own IRC client, to avoid those problems. The first thing I realized is, it is not a good idea to think about writing your own IRC client. Danger will robinson.. So, let's generalize. This blog post is not about designing an IRC client, but about exploring simpler ways that something like an IRC client might handle its UI, and perhaps designing something general-purpose that could be used by someone else to build an IRC client, or be mashed up with an existing IRC client. What any modern IRC client needs to do is display various channels to the user. Maybe more than one channel should be visible at a time in some kind of window, but often the user will have lots of available channel and only want to see a few of them at a time. So there needs to be an interface for picking which channel(s) to display, and if multiple windows are shown, for arranging the windows. Often that interface also indicates when there is activity on a channel. The most recent messages from the channel are displayed. There should be a way to scroll back to see messages that have already scrolled by. There needs to be an interface for sending a message to a channel. Finally, a list of users in the channel is often desired. Modern IRC clients implement their own UI for channel display, windowing, channel selection, activity notification, scrollback, message entry, and user list. Even the IRC clients that run in a terminal include all of that. But how much of that do they need to implement, really? Suppose the user has a tabbed window manager, that can display virtual terminals. The terminals can set their title, and can indicate when there is activity in the terminal. Then an IRC client could just open a bunch of terminals, one per channel. Let the window manager handle channel selection, windowing (naturally), and activity notification. For scrollback, the IRC client can use the terminal's own scrollback buffer, so the terminal's regular scrollback interface can be used. This is slightly tricky; can't use the terminal's alternate display, and have to handle the UI for the message entry line at the bottom. That's all the UI an IRC client needs (except for the user list), and most of that is already implemented in the window manager and virtual terminal. So that's an elegant way to make an IRC client without building much new UI at all. But, unfortunately, most of us don't use tabbed window managers (or tabbed terminals). Such an IRC client, in a non-tabbed window manager, would be a many-windowed mess. Even in a tabbed window manager, it might be annoying to have so many windows for one program. So we need fewer windows. Let's have one channel list window, and one channel display window. There could also be a user list window. And there could be a way to open additional, dedicated display windows for channels, but that's optional. All of these windows can be seperate virtual terminals. A potential problem: When changing the displayed channel, it needs to output a significant number of messages for that channel, so that the scrollback buffer gets populated. With a large number of lines, that can be too slow to feel snappy. In some tests, scrolling 10 thousand lines was noticiably slow, but scrolling 1 thousand lines happens fast enough not to be noticiable. (Terminals should really be faster at scrolling than this, but they're still writing scrollback to unlinked temp files.. sigh!) An IRC client that uses multiple cooperating virtual terminals, needs a way to start up a new virtual terminal displaying the current channel. It could run something like this: 
x-terminal-emulator -e the-irc-client --display-current-channel
That would communicate with the main process via a unix socket to find out what to display. Or, more generally: 
x-terminal-emulator -e connect-pty /dev/pts/9
connect-pty would simply connect a pty device to the terminal, relaying IO between them. The calling program would allocate the pty and do IO to it. This may be too general to be optimal though. For one thing, I think that most curses libraries have a singleton terminal baked into them, so it might be hard to have a single process control cursors on multiple pty's. And, it might be innefficient to feed that 1 thousand lines of scrollback through the pty and copy it to the terminal. Less general than that, but still general enough to not involve writing an IRC client, would be a library that handled the irc-client-like channel display, with messages scrolling up the terminal (and into the scrollback buffer), a input area at the bottom, and perhaps a few other fixed regions for status bars and etc. Oh, I already implemented that! In concurrent-output, over a year ago: a tiling region manager for the console I wonder what other terminal applications could be simplified/improved by using multiple terminals? One that comes to mind is mutt, which has a folder list, a folder view, and an email view, that all are shoehorned, with some complexity, into a single terminal.

24 December 2016

Russ Allbery: rra-c-util 6.2

This is my general collection of utility functions, standard tests, and portability code, mostly for C but also including a fair bit of Perl these days. This release improves probing for GCC and Clang warning flags (thanks to Guillem Jover for a good hint there), adds a new is_file_contents test function to the Perl Test::RRA module for comparing whole files, and adds a new test that scans for non-https eyrie.org URLs in my documentation. It also fixes some issues with the Perl strictness test caused by not requiring a new enough version of Test::Strict. You can get the latest release from the rra-c-util distribution page.

2 December 2016

Shirish Agarwal: Air Congestion and Politics

Confession time first I am not a frequent flyer at all. My first flight was in early late 2006. It was a 2 hour flight from Bombay (BOM) to Bengaluru (formerly Bangalore, BLG) . I still remember the trepidation, the nervousness and excitement the first time I took to air. I still remember the flight very vividly, It was a typical humid day for Bombay/Mumbai and we (me and a friend) had gone to Sahar (the domestic airport) to take the flight in the evening. Before starting the sky had turned golden-orange and I was wondering how I would feel once I would be in air.We started at around 20:00 hours in the evening and as it was a clear night were able to see the Queen s necklace (Marine Drive) in all her glory. The photographs on the wikipedia page don t really do justice to how beautiful the whole boulevard looks at night, especially how it looks from up there. While we were seeing, it seemed the pilot had actually banked at 45 degrees angle so we can have the best view of the necklace OR maybe the pilot wanted to take a photo OR ME being in overdrive (like Robin Williams, the Russian immigrant in Moscow on the Hudson experiences the first time he goes to the mall ;)) In either way, this would be an experience I would never forget till the rest of my life. I remember I didn t move an inch (even to go the loo) as I didn t want to let go of the whole experience. While I came back after 3-4 days, I still remember re-experiencing/re-imagining the flights for a whole month each time I went to sleep. While I can t say it has become routinised, but have been lucky to have the opportunity to fly domestic around the country primarily for work. After the initial romanticism wears off, you try and understand the various aspects of the flight which are happening around you. These experiences are what lead to file/share today s blog post. Yesterday, Ms. Mamata Banerjee, one of the leaders of the Opposition cried wolf because the Aircraft was circling the Airport. Because she is the Chief Minister she feels she should have got precedent or at least that seems to be the way the story unfolded on TV. I have been about 15-20 times on flight in the last decade for work or leisure. Almost all the flights I have been, it has been routine that the flights fly around the Airport for 15-20 minutes before landing. This is routine . I have seen Airlines being stacked (remember the scene from Die Hard 2 where Holly Mclane, John Mclane s wife looks at different aircraft at different altitudes from her window seat) this is what an Airport has to do when it doesn t have enough runaways. In fact just read few days back MIAL is going for an emergency expansion as they weren t expecting as many passengers as they did this year as well as last. In fact the same day there was a near-miss between two aircraft in Mumbai airport itself. Because of Ms. Mamata s belligerence, this story didn t even get a mention in the TV mainstream media. The point I wanna underscore is that this is a fact of life and not just in India, world-over it seems hubs are being busier than ever, for instance Heathrow has been also a busy bee and they will to rework air operations as per a recent article . In India, Kolkata is also one of the busier airports . If anything, I hope it teaches her the issues that plague most Indian airports and she works with the Government in Center so the Airport can expand more. They just got a new terminal three years back. It is for these issues that the Indian Government has come with the Regional Connectivity Scheme . Lastly, a bit of welcome news to people thinking to visit India, the Govt. of the day is facilitating easier visa norms to increase tourism and trade to India. Hope this is beneficial to all and any Debian Developers who wanna come visit India I do hope that we also do get reciprocity from those countries as well.
Filed under: Miscellenous Tagged: # Domestic Flights, #Air Congestion, #Airport Expansion, #Kolkata, #near-miss, #Visa for tourists

3 November 2016

Jan Wagner: Container Orchestration Thoughts

Container Orchestration ThoughtsSince some time everybody (read developer) want to run his new microservice stacks in containers. I can understand that building and testing an application is important for developers.
One of the benefits of containers is, that developer (in theory) can put their new version of applications into production on their own. This is the point where operations is affected and operations needs to evaluate, if that might evolve into better workflow. For yolo^WdevOps people there are some challenges that needs to be solved, or at least mitigated, when things needs to be done in large(r) scale.

Orchestration Engine Running Docker, which is actual the most preferred container solution, on a single host with docker command line client is something you can do, but there you leave the gap between dev and ops.

UI For Docker Since some time there is UI For Docker available for visualizing and managing containers on a single docker node. It's pretty awesome and the best feature so far is the Container Network view, which also shows the linked container. Container Orchestration Thoughts

Portainer Portainer is pretty new and it can be deployed as easy as UI For Docker. But the (first) great advantage: it can handle Docker Swarm. Beside that it has many other great features. Container Orchestration Thoughts

Rancher Rancher describes themselves as 'container management platform' that 'supports and manages all of your Kubernetes, Mesos, and Swarm clusters'. This is great because this are all of the relevant docker cluster orchestrations at the market actually. Container Orchestration Thoughts For the use cases, we are facing, Kubernetes and Mesos seems both like bloated beasts. Usman Ismail has written a really good comparison of Orchestration Engine options which goes into details. Container Orchestration Thoughts

Docker Swarm As there is actually no clear defacto standard/winner of the (container) orchestration wars, I would prevent to be in a vendor lock-in situation (yet). Docker swarm seems to be evolving and is getting more nice features other competitors doesn't provide.
Due the native integration into the docker framework and great community I believe Docker Swarm will be the Docker Orchestration of the choice on the long run. This should be supported by Rancher 1.2 which is not released yet.
From this point of view it looks very reasonable that Docker Swarm in combination with Rancher (1.2) might be a good strategy to maintain your container farms in the future. If you think to put Docker Swarm into production in the actual state, I recommend to read Docker swarm mode: What to know before going live on production by Panjamapong Sermsawatsri.

Persistent Storage While it is a best practice to use data volume container these days, providing persistent storage across multiple hosts for shared volumes seems to be tricky. In theory you can mount a shared-storage volume as a data volume and there are several volume plugins which supports shared storage. For example you can use the convoy plugin which gives you:
  • thin provisioned volumes
  • snapshots of volumes
  • backup of snapshots
  • restore volumes
As backend you can use:
  • Device Mapper
  • Virtual File System(VFS)/Network File System(NFS)
  • Amazon Elastic Block Store(EBS)
The good thing is, that convoy is integrated into Rancher. For more information I suggest to read Setting Up Shared Volumes with Convoy-NFS, which also mentions some limitations. If you want test Persistent Storage Service, Rancher provides some documentation. Actually I did not evaluate shared-storage volumes yet, but I don't see a solution I would love to use in production (at least on-premise) without strong downsides. But maybe things will go further and there might be a great solution for this caveats in the future.

Keeping base images up-to-date Since some time there are many projects that tries to detect security problems in your container images in several ways.
Beside general security considerations you need to deal somehow with issues in your base images that you build your applications on. Of course, even if you know you have a security issue in your application image, you need to fix it, which depends on the way how you based your application upon.

Ways to base your application image
  • You can build your application image entire from scratch, which leaves all the work to your development team and I wouldn't recommend it that way.
  • You also can create one (or more) intermediate image(s) that will be used by your development team.
  • The development team might ground their work on images in public available or private (for example the one bundled to your gitlab CI/CD solution) registries.

Whats the struggle with the base image? If you are using images being not (well) maintained by other people, you have to wait for them to fix your base image. Using external images might also lead into trust problems (can you trust those people in general?).
In an ideal world, your developers have always fresh base images with fixed security issues. This can probably be done by rebuilding every intermediate image periodically or when the base image changes.

Paradigm change Anyway, if you have a new application image available (with no known security issues), you need to deploy it to production. This is summarized by Jason McKay in his article Docker Security: How to Monitor and Patch Containers in the Cloud:
To implement a patch, update the base image and then rebuild the application image. This will require systems and development teams to work closely together.
So patching security issues in the container world changes workflow significant. In the old world operation teams mostly rolled security fixes for the base systems independent from development teams.
Now hitting containers the production area this might change things significant.

Bringing updated images to production Imagine your development team doesn't work steady on a project, cause the product owner consider it feature complete. The base image is provided (in some way) consistently without security issues. The application image is build on top of that automatically on every update of the base image.
How do you push in such a scenario the security fixes to production? From my point of view you have two choices:
  • Let the development team require to test the resulting application image and put it into production
  • Push the new application image without review by the development team into production
The first scenario might lead into a significant delay until the fixes hit production created by the probably infrequent work of the development team. The latter one brings your security fixes early to production by the notable higher risk to break your application. This risk can be reduced by implementing massive tests into CI/CD pipelines by the development team. Rolling updates provided by Docker Swarm might also reduce the risk of ending with a broken application. When you are implementing an update process of your (application) images to production, you should consider Watchtower that provides Automatic Updates for Docker Containers.

Conclusion Not being a product owner or the operations part of an application that is facing a widely adopted usage that would compensate the actual tradeoffs we are still facing I tend not to move large scale production projects into a container environment.
This means not that this might be a bad idea for others, but I'd like to sort out some of the caveats before. I'm still interested to put smaller projects into production, being not scared to reimplement or move them on a new stack.
For smaller projects with a small number of hosts Portainer looks not bad as well as Rancher with the Cattle orchestration engine if you just want to manage a couple of nodes. Things are going to be interesting if Rancher 1.2 supports Docker swarm cluster out of the box. Let's see what the future will bring us to the container world and how to make a great stack out of it.

Update I suggest to read Docker in Production: A History of Failure and the answer Docker in Production: A retort to understand the actual challenges when running Docker in larger scale production environments.

8 July 2016

Mike Hommey: Are all integer overflows equal?

Background: I ve been relearning Rust (more about that in a separate post, some time later), and in doing so, I chose to implement the low-level parts of git (I ll touch the why in that separate post I just promised). Disclaimer: It s friday. This is not entirely(?) a serious post. So, I was looking at Documentation/technical/index-format.txt, and saw:
32-bit number of index entries.
What? The index/staging area can t handle more than ~4.3 billion files? There I was, writing Rust code to write out the index. 
try!(out.write_u32::<NetworkOrder>(self.entries.len()));
(For people familiar with the byteorder crate and wondering what NetworkOrder is, I have a use byteorder::BigEndian as NetworkOrder) And the Rust compiler rightfully barfed: 
error: mismatched types:
 expected  u32 ,
    found  usize  [E0308]
And there I was, wondering: mmmm should I just add as u32 and silently truncate or hey what does git do? And it turns out, git uses an unsigned int to track the number of entries in the first place, so there is no truncation happening. Then I thought but what happens when cache_nr reaches the max? Well, it turns out there s only one obvious place where the field is incremented. What? Holy coffin nails, Batman! No overflow check? Wait a second, look 3 lines above that: 
ALLOC_GROW(istate->cache, istate->cache_nr + 1, istate->cache_alloc);
Yeah, obviously, if you re incrementing cache_nr, you already have that many entries in memory. So, how big would that array be? 
        struct cache_entry **cache;
So it s an array of pointers, assuming 64-bits pointers, that s ~34.3 GB. But, all those cache_nr entries are in memory too. How big is a cache entry? 
struct cache_entry  
        struct hashmap_entry ent;
        struct stat_data ce_stat_data;
        unsigned int ce_mode;
        unsigned int ce_flags;
        unsigned int ce_namelen;
        unsigned int index;     /* for link extension */
        unsigned char sha1[20];
        char name[FLEX_ARRAY]; /* more */
 ;
So, 4 ints, 20 bytes, and as many bytes as necessary to hold a path. And two inline structs. How big are they? 

struct hashmap_entry  
        struct hashmap_entry *next;
        unsigned int hash;
 ;
struct stat_data  
        struct cache_time sd_ctime;
        struct cache_time sd_mtime;
        unsigned int sd_dev;
        unsigned int sd_ino;
        unsigned int sd_uid;
        unsigned int sd_gid;
        unsigned int sd_size;
 ;
Woohoo, nested structs. 
struct cache_time  
        uint32_t sec;
        uint32_t nsec;
 ;
So all in all, we re looking at 1 + 2 + 2 + 5 + 4 32-bit integers, 1 64-bits pointer, 2 32-bits padding, 20 bytes of sha1, for a total of 92 bytes, not counting the variable size for file paths. The average path length in mozilla-central, which only has slightly over 140 thousands of them, is 59 (including the terminal NUL character). Let s conservatively assume our crazy repository would have the same average, making the average cache entry 151 bytes. But memory allocators usually allocate more than requested. In this particular case, with the default allocator on GNU/Linux, it s 156 (weirdly enough, it s 152 on my machine). 156 times 4.3 billion 670 GB. Plus the 34.3 from the array of pointers: 704.3 GB. Of RAM. Not counting the memory allocator overhead of handling that. Or all the other things git might have in memory as well (which apparently involves a hashmap, too, but I won t look at that, I promise). I think one would have run out of memory before hitting that integer overflow. Interestingly, looking at Documentation/technical/index-format.txt again, the on-disk format appears smaller, with 62 bytes per file instead of 92, so the corresponding index file would be smaller. (And in version 4, paths are prefix-compressed, so paths would be smaller too). But having an index that large supposes those files are checked out. So let s say I have an empty ext4 file system as large as possible (which I m told is 2^60 bytes (1.15 billion gigabytes)). Creating a small empty ext4 tells me at least 10 inodes are allocated by default. I seem to remember there s at least one reserved for the journal, there s the top-level directory, and there s lost+found ; there apparently are more. Obviously, on that very large file system, We d have a git repository. git init with an empty template creates 9 files and directories, so that s 19 more inodes taken. But git init doesn t create an index, and doesn t have any objects. We d thus have at least one file for our hundreds of gigabyte index, and at least 2 who-knows-how-big files for the objects (a pack and its index). How many inodes does that leave us with? The Linux kernel source tells us the number of inodes in an ext4 file system is stored in a 32-bits integer. So all in all, if we had an empty very large file system, we d only be able to store, at best, 2^32 22 files And we wouldn t even be able to get cache_nr to overflow. while following the rules. Because the index can keep files that have been removed, it is actually possible to fill the index without filling the file system. After hours (days? months? years? decades?*) of running 
seq 0 4294967296   while read i; do touch $i; git update-index --add $i; rm $i; done
One should be able to reach the integer overflow. But that d still require hundreds of gigabytes of disk space and even more RAM. Ok, it s actually much faster to do it hundreds of thousand files at a time, with something like: 
seq 0 100000 4294967296   while read i; do j=$(seq $i $(($i + 99999))); touch $j; git update-index --add $j; rm $j; done
At the rate the first million files were added, still assuming a constant rate, it would take about a month on my machine. Considering reading/writing a list of a million files is a thousand times faster than reading a list of a billion files, assuming linear increase, we re still talking about decades, and plentiful RAM. Fun fact: after leaving it run for 5 times as much as it had run for the first million files, it hasn t even done half more One could generate the necessary hundreds-of-gigabytes index manually, that wouldn t be too hard, and assuming it could be done at about 1 GB/s on a good machine with a good SSD, we d be able to craft a close-to-explosion index within a few minutes. But we d still lack the RAM to load it. So, here is the open question: should I report that integer overflow? Wow, that was some serious procrastination. Edit: Epilogue: Actually, oops, there is a separate integer overflow on the reading side that can trigger a buffer overflow, that doesn t actually require a large index, just a crafted header, demonstrating that yes, not all integer overflows are equal.

15 June 2016

Russ Allbery: Review: Matter

Review: Matter, by Iain M. Banks
Publisher: Orbit
Copyright: February 2008
ISBN: 0-316-00536-3
Format: Hardcover
Pages: 593
Sursamen is an Arithmetic, Mottled, Disputed, Multiply Inhabited, Multi-million Year Safe, and Godded Shellworld. It's a constructed world with multiple inhabitable levels, each lit by thermonuclear "suns" on tracks, each level supported above the last by giant pillars. Before the recorded history of the current Involved Species, a culture called the Veil created the shellworlds with still-obscure technology for some unknown purpose, and then disappeared. Now, they're inhabited by various transplants and watched over by a hierarchy of mentor and client species. In the case of Sursamen, both the Aultridia and the Oct claim jurisdiction (hence "Disputed"), and are forced into an uneasy truce by the Nariscene, a more sophisticated species that oversees them both. On Sursamen, on level eight to be precise, are the Sarl, a culture with an early industrial level of technology in the middle of a war of conquest to unite their level (and, they hope, the next level down). Their mentors are the Oct, who claim descendance from the mysterious Veil. The Deldeyn, the next level down, are mentored by the Aultridia, a species that evolved from a parasite on Xinthian Tensile Aranothaurs. Since a Xinthian, treated by the Sarl as a god, lives in the heart of Sursamen (hence "Godded"), tensions between the Sarl and the Aultridians run understandably high. The ruler of the Sarl had three sons and a daughter. The oldest was killed by the people he is conquering as Matter starts. The middle son is a womanizer and a fop who, as the book opens, watches a betrayal that he's entirely unprepared to deal with. The youngest is a thoughtful, bookish youth pressed into a position that he also is not well-prepared for. His daughter left the Sarl, and Sursamen itself, fifteen years previously. Now, she's a Special Circumstances agent for the Culture. Matter is the eighth Culture novel, although (like most of the series) there's little need to read the books in any particular order. The introduction to the Culture here is a bit scanty, so you'll have more background and understanding if you've read the previous novels, but it doesn't matter a great deal for the story. Sharp differences in technology levels have turned up in previous Culture novels (although the most notable example is a minor spoiler), but this is the first Culture novel I recall where those technological differences were given a structure. Usually, Culture novels have Special Circumstances meddling in, from their perspective, "inferior" cultures. But Sursamen is not in Culture space or directly the Culture's business. The Involved Species that governs Sursamen space is the Morthanveld: an aquatic species roughly on a technology level with the Culture themselves. The Nariscene are their client species; the Oct and Aultridia are, in turn, client species (well, mostly) of the Nariscene, while meddling with the Sarl and Deldeyn. That part of this book reminded me of Brin's Uplift universe. Banks's Involved Species aren't the obnoxious tyrants of Brin's universe, and mentoring doesn't involve the slavery of the Uplift universe. But some of the politics are a bit similar. And, as with Uplift, all the characters are aware, at least vaguely, of the larger shape of galactic politics. Even the Sarl, who themselves have no more than early industrial technology. When Ferbin flees the betrayal to try to get help, he ascends out of the shellworld to try to get assistance from an Involved species, or perhaps his sister (which turns out to be the same thing). Banks spends some time here, mostly through Ferbin and his servant (who is one of the better characters in this book), trying to imagine what it would be like to live in a society that just invented railroads while being aware of interstellar powers that can do practically anything. The plot, like the world on which it's set, proceeds on multiple levels. There is court intrigue within the Sarl, war on their level and the level below, and Ferbin's search for support and then justice. But the Sarl live in an artifact with some very mysterious places, including the best set piece in the book: an enormous waterfall that's gradually uncovering a lost city on the level below the Sarl, and an archaeological dig that proceeds under the Deldeyn and Sarl alike. Djan Seriy decides to return home when she learns of events in Sarl, originally for reasons of family loyalty and obligation, but she's a bit more in touch with the broader affairs of the galaxy, including the fact that the Oct are acting very strangely. There's something much greater at stake on Sursamen than tedious infighting between non-Involved cultures. As always with Banks, the set pieces and world building are amazing, the scenery is jaw-dropping, and I have some trouble warming to the characters. Dramatic flights across tower-studded landscapes seeking access to forbidden world-spanning towers largely, but don't entirely, make up for not caring about most of the characters for most of the book. This did change, though: although I never particularly warmed to Ferbin, I started to like his younger brother, and I really liked his sister and his servant by the end of the book. Unfortunately, the end of Matter is, if not awful, at least exceedingly abrupt. As is typical of Banks, we get a lot of sense of wonder but not much actual explanation, and the denouement is essentially nonexistent. (There is a coy epilogue hiding after the appendices, but it mostly annoyed me and provides only material for extrapolation about the characters.) Another SF author would have written a book about the Xinthian, the Veil, the purpose of the shellworlds, and the deep history of the galaxy. I should have known going in that Banks isn't that sort of SF author, but it was still frustrating. Still, Banks is an excellent writer and this is a meaty, complex, enjoyable story with some amazing moments of wonder and awe. If you like Culture novels in general, you will like this. If you like set-piece-heavy SF on a grand scale, such as Alastair Reynolds or Kim Stanley Robinson, you probably like this. Recommended. Rating: 8 out of 10

10 April 2016

Russ Allbery: Largish haul

Let's see if I can scrounge through all of my now-organized directories of ebooks and figure out what I haven't recorded here yet. At least the paper books make that relatively easy, since I don't shelve them until I post them. (Yeah, yeah, I should actually make a database.) Hugh Aldersey-Williams Periodic Tales (nonfiction)
Sandra Ulbrich Almazan SF Women A-Z (nonfiction)
Radley Balko Rise of the Warrior Cop (nonfiction)
Peter V. Brett The Warded Man (sff)
Lois McMaster Bujold Gentleman Jole and the Red Queen (sff)
Fred Clark The Anti-Christ Handbook Vol. 2 (nonfiction)
Dave Duncan West of January (sff)
Karl Fogel Producing Open Source Software (nonfiction)
Philip Gourevitch We Wish to Inform You That Tomorrow We Will Be Killed With Our Families (nonfiction)
Andrew Groen Empires of EVE (nonfiction)
John Harris @ Play (nonfiction)
David Hellman & Tevis Thompson Second Quest (graphic novel)
M.C.A. Hogarth Earthrise (sff)
S.L. Huang An Examination of Collegial Dynamics... (sff)
S.L. Huang & Kurt Hunt Up and Coming (sff anthology)
Kameron Hurley Infidel (sff)
Kevin Jackson-Mead & J. Robinson Wheeler IF Theory Reader (nonfiction)
Rosemary Kirstein The Lost Steersman (sff)
Rosemary Kirstein The Language of Power (sff)
Merritt Kopas Videogames for Humans (nonfiction)
Alisa Krasnostein & Alexandra Pierce (ed.) Letters to Tiptree (nonfiction)
Mathew Kumar Exp. Negatives (nonfiction)
Ken Liu The Grace of Kings (sff)
Susan MacGregor The Tattooed Witch (sff)
Helen Marshall Gifts for the One Who Comes After (sff collection)
Jack McDevitt Coming Home (sff)
Seanan McGuire A Red-Rose Chain (sff)
Seanan McGuire Velveteen vs. The Multiverse (sff)
Seanan McGuire The Winter Long (sff)
Marc Miller Agent of the Imperium (sff)
Randal Munroe Thing Explainer (graphic nonfiction)
Marguerite Reed Archangel (sff)
J.K. Rowling Harry Potter: The Complete Collection (sff)
K.J. Russell Tides of Possibility (sff anthology)
Robert J. Sawyer Starplex (sff)
Bruce Schneier Secrets & Lies (nonfiction)
Mike Selinker (ed.) The Kobold Game to Board Game Design (nonfiction)
Douglas Smith Chimerascope (sff collection)
Jonathan Strahan Fearsome Journeys (sff anthology)
Nick Suttner Shadow of the Colossus (nonfiction)
Aaron Swartz The Boy Who Could Change the World (essays)
Caitlin Sweet The Pattern Scars (sff)
John Szczepaniak The Untold History of Japanese Game Developers I (nonfiction)
John Szczepaniak The Untold History of Japanese Game Developers II (nonfiction)
Jeffrey Toobin The Run of His Life (nonfiction)
Hayden Trenholm Blood and Water (sff anthology)
Coen Teulings & Richard Baldwin (ed.) Secular Stagnation (nonfiction)
Ursula Vernon Book of the Wombat 2015 (graphic nonfiction)
Ursula Vernon Digger (graphic novel) Phew, that was a ton of stuff. A bunch of these were from two large StoryBundle bundles, which is a great source of cheap DRM-free ebooks, although still rather hit and miss. There's a lot of just fairly random stuff that's been accumulating for a while, even though I've not had a chance to read very much. Vacation upcoming, which will be a nice time to catch up on reading.

14 March 2016

Bits from Debian: New Debian Developers and Maintainers (January and February 2016)

The following contributors got their Debian Developer accounts in the last two months: The following contributors were added as Debian Maintainers in the last two months: Congratulations!

1 March 2016

Ritesh Raj Sarraf: Linux Power Savings 2016

Having moved to a new place, now at times, I also have to deal with power outages. As heat increases, the power outages will be much longer and more frequent. So much, that UPS and Power Inverters run out. Such are ideal times to measure idle power consumption for my laptop. Here's what my default (and idle) OS looks like. It should be standard to most "typical" users. Some minor odds could be apport, dnsmasq, and maybe, tor. Also, thanks to systemd, there are some native services which are now converted to socket activation based ones. There's also Laptop Mode Tools for userspace power savings. But huge thanks to Intel's work in making the kernel more power efficient. With things like Runtime PM and Intel P-State, we can now define, default and aggressive power-savings modes, helping conserve more power. We still have challenges with things like binary-only drivers; and also stock GPL drivers but with an external firmware blob. There have been many instances when my previous machines (which had both) ran into severe power drain. The challenge with power savings is that it is not just the job of the kernel. All components, kernel and userspace, need to be made aware of, and work in unison. Otherwise, one ill behaving web browser tab can be the cause of power drain. P-State driver seems to already be doing some scale down of CPU resources. And with CGroups maturing moe I think we'll eventually have all Desktop Environments making use of it. 
systemd-+-ModemManager-+- gdbus 
                        - gmain 
         -NetworkManager-+- gdbus 
                          - gmain 
         -accounts-daemon-+- gdbus 
                           - gmain 
         -agetty
         -apport-notifyd
         -atd
         -avahi-daemon---avahi-daemon
         -bluetoothd
         -colord-+- gdbus 
                  - gmain 
         -cron
         -dbus-daemon
         -dnsmasq
         -evolution-+- book-client-dbu 
                     - dconf worker 
                     -7*[ evolution ]
                     - gdbus 
                     - gmain 
         -gdm3-+-gdm-session-wor-+-gdm-x-session-+-Xorg
                                                  -gnome-session-b-+-gnome-settings--+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                                      - pool 
                                                                                      - threaded-ml 
                                                                    -gnome-shell-+-ibus-daemon-+-ibus-dconf-+- dconf worker 
                                                                                                             - gdbus 
                                                                                                             - gmain 
                                                                                                -ibus-engine-sim-+- gdbus 
                                                                                                                  - gmain 
                                                                                                - gdbus 
                                                                                                - gmain 
                                                                                  - JS GC Helper 
                                                                                  - JS Sour~ Thread 
                                                                                  - dconf worker 
                                                                                  - gdbus 
                                                                                  - gmain 
                                                                                  - threaded-ml 
                                                                    - dconf worker 
                                                                    - gdbus 
                                                                    - gmain 
                                                  - gdbus 
                                                  - gmain 
                                  - gdbus 
                                  - gmain 
                -gdm-session-wor-+-gdm-x-session-+-Xorg
                                                  -gnome-session-b-+-evolution-alarm-+- cal-client-dbus 
                                                                                      - dconf worker 
                                                                                      - evolution-alarm 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                    -gnome-settings--+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                                      - pool 
                                                                                      - threaded-ml 
                                                                    -gnome-shell-+-2*[redshift-+- gdbus ]
                                                                                                - gmain ]
                                                                                  - JS GC Helper 
                                                                                  - JS Sour~ Thread 
                                                                                  - dconf worker 
                                                                                  - gdbus 
                                                                                  - gmain 
                                                                                  - threaded-ml 
                                                                    -parcimonie
                                                                    -parcimonie-appl-+- gdbus 
                                                                                      - gmain 
                                                                    -python3---libinput-debug-
                                                                    -ssh-agent
                                                                    -tracker-extract-+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                                      -10*[ pool ]
                                                                    -tracker-miner-a-+- gdbus 
                                                                                      - gmain 
                                                                    -tracker-miner-f-+- dconf worker 
                                                                                      - gdbus 
                                                                                      - gmain 
                                                                    -tracker-miner-u-+- gdbus 
                                                                                      - gmain 
                                                                    - dconf worker 
                                                                    - gdbus 
                                                                    - gmain 
                                                  - gdbus 
                                                  - gmain 
                                  - gdbus 
                                  - gmain 
                - gdbus 
                - gmain 
         -geoclue-+- gdbus 
                   - gmain 
         -gnome-keyring-d-+- gdbus 
                           - gmain 
                           - timer 
         -gpg-agent
         -gsd-printer-+- gdbus 
                       - gmain 
         -ibus-daemon-+-ibus-dconf-+- dconf worker 
                                    - gdbus 
                                    - gmain 
                       -ibus-engine-sim-+- gdbus 
                                         - gmain 
                       - gdbus 
                       - gmain 
         -ibus-x11-+- gdbus 
                    - gmain 
         -iio-sensor-prox-+- gdbus 
                           - gmain 
         -inetd---leafnode
         -libvirtd---15*[ libvirtd ]
         -lvmetad
         -master-+-pickup
                  -qmgr
                  -tlsmgr
         -mcelog
         -minissdpd
         -mount.ntfs
         -onboard-+- dconf worker 
                   - gdbus 
                   - gmain 
         -packagekitd-+- gdbus 
                       - gmain 
         -polkitd-+- gdbus 
                   - gmain 
         -2*[pulseaudio-+-gconf-helper]
                         - alsa-sink-ALC23 ]
                         - alsa-sink-HDMI  ]
                         - alsa-sink-pcspe ]
                         - alsa-source-ALC ]
         -rtkit-daemon---2*[ rtkit-daemon ]
         -systemd-+-(sd-pam)
                   -at-spi-bus-laun-+-dbus-daemon
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -at-spi2-registr-+- gdbus 
                                     - gmain 
                   -dbus-daemon
                   -dconf-service-+- gdbus 
                                   - gmain 
                   -goa-daemon-+- dconf worker 
                                - gdbus 
                                - gmain 
                   -goa-identity-se-+- gdbus 
                                     - gmain 
                                     - pool 
                   -gvfs-afc-volume-+- gdbus 
                                     - gmain 
                                     - gvfs-afc-volume 
                   -gvfs-goa-volume-+- gdbus 
                                     - gmain 
                   -gvfs-gphoto2-vo-+- gdbus 
                                     - gmain 
                   -gvfs-mtp-volume-+- gdbus 
                                     - gmain 
                   -gvfs-udisks2-vo-+- gdbus 
                                     - gmain 
                   -gvfsd-+- gdbus 
                           - gmain 
                   -gvfsd-fuse-+- gdbus 
                                - gmain 
                                - gvfs-fuse-sub 
                                -2*[ gvfsd-fuse ]
                   -gvfsd-metadata-+- gdbus 
                                    - gmain 
                   -mission-control-+- dconf worker 
                                     - gdbus 
                                     - gmain 
         -systemd-+-(sd-pam)
                   -at-spi-bus-laun-+-dbus-daemon
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -at-spi2-registr-+- gdbus 
                                     - gmain 
                   -dbus-daemon
                   -gconfd-2
         -systemd-+-(sd-pam)
                   -at-spi-bus-laun-+-dbus-daemon
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -at-spi2-registr-+- gdbus 
                                     - gmain 
                   -dbus-daemon
                   -dconf-service-+- gdbus 
                                   - gmain 
                   -evolution-addre-+-2*[evolution-addre-+- dconf worker ]
                                                          - evolution-addre ]
                                                          - gdbus ]
                                                          - gmain ]
                                     - dconf worker 
                                     - evolution-addre 
                                     - gdbus 
                                     - gmain 
                   -evolution-calen-+-evolution-calen-+- dconf worker 
                                                       - evolution-calen 
                                                       - gdbus 
                                                       - gmain 
                                                       -6*[ pool ]
                                     -evolution-calen-+- book-client-dbu 
                                                       - dconf worker 
                                                       - evolution-calen 
                                                       - gdbus 
                                                       - gmain 
                                                       - pool 
                                     -2*[evolution-calen-+- dconf worker ]
                                                          - evolution-calen ]
                                                          - gdbus ]
                                                          - gmain ]
                                     - dconf worker 
                                     - evolution-calen 
                                     - gdbus 
                                     - gmain 
                   -evolution-sourc-+- dconf worker 
                                     - gdbus 
                                     - gmain 
                   -gconfd-2
                   -gnome-shell-cal-+- cal-client-dbus 
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                                     - gnome-shell-cal 
                   -gnome-terminal--+-bash---pstree
                                     -bash
                                     - dconf worker 
                                     - gdbus 
                                     - gmain 
                   -goa-daemon-+- dconf worker 
                                - gdbus 
                                - gmain 
                   -goa-identity-se-+- gdbus 
                                     - gmain 
                                     - pool 
                   -gpaste-daemon-+- dconf worker 
                                   - gdbus 
                                   - gmain 
                   -gvfs-afc-volume-+- gdbus 
                                     - gmain 
                                     - gvfs-afc-volume 
                   -gvfs-goa-volume-+- gdbus 
                                     - gmain 
                   -gvfs-gphoto2-vo-+- gdbus 
                                     - gmain 
                   -gvfs-mtp-volume-+- gdbus 
                                     - gmain 
                   -gvfs-udisks2-vo-+- gdbus 
                                     - gmain 
                   -gvfsd-+- gdbus 
                           - gmain 
                   -gvfsd-burn-+- gdbus 
                                - gmain 
                   -gvfsd-dnssd-+- gdbus 
                                 - gmain 
                   -gvfsd-fuse-+- gdbus 
                                - gmain 
                                - gvfs-fuse-sub 
                                -2*[ gvfsd-fuse ]
                   -gvfsd-metadata-+- gdbus 
                                    - gmain 
                   -gvfsd-network-+- dconf worker 
                                   - gdbus 
                                   - gmain 
                   -gvfsd-trash-+- gdbus 
                                 - gmain 
                   -mission-control-+- dconf worker 
                                     - gdbus 
                                     - gmain 
                   -nautilus-+-gedit
                              - dconf worker 
                              - gdbus 
                              - gmain 
                   -tracker-store-+- dconf worker 
                                   - gdbus 
                                   - gmain 
                                   -4*[ pool ]
         -systemd-journal
         -systemd-logind
         -systemd-network
         -systemd-timesyn--- sd-resolve 
         -systemd-udevd
         -thermald--- thermald 
         -tor
         -udisksd-+- cleanup 
                   - gdbus 
                   - gmain 
                   - probing-thread 
         -upowerd-+- gdbus 
                   - gmain 
         -wpa_supplicant
So I used the standard power measurement tool, powertop. Below is a screenshot of PowerTop claiming 9 - 10 hours of possible battery backup. This is idle state. "Idle" would mean when the user is not interacting with the machine. When under basic usage (email, web), the actual battery backup I've sensed is somewhere around 6-7 hrs, which is still good from what we got a couple years ago. There's one oddity in the screenshot though. The estimated power consumption reported is 4.55W, where as, the power consumption of just the wifi card is reported 6.17W. That too, when the wifi was disconnected. This reminded me of the bug report against PowerTop, which is now more than 2 years old. We've still not concluded in that bug report, so if anyone can shed some light on PowerTop's reporting, please do share it in the bug report. Note: Please ignore the terminal's title in the screenshot. That is wrong and I've not bothered to figure out why it is displaying the title name constructed from an outdated session, which is already terminated.

Categories:

Keywords:

Like:

14 October 2015

Russ Allbery: Review: Firebird

Review: Firebird, by Jack McDevitt
Series: Alex Benedict #6
Publisher: Ace
Copyright: November 2011
Printing: October 2012
ISBN: 1-937007-80-4
Format: Mass market
Pages: 357
Firebird is the sixth book in the Alex Benedict series about an antique hunter in the far future (albeit a far future that looks exactly like 1960s suburbia, but with spaceships). This is a very episodic series, though, and I don't think it would be hurt much by starting in the middle. There are references to earlier investigations, but they're fleeting, and I often didn't map them to remembered plots even though I've read the whole series to this point. The investigation in this book starts with the heir to an estate coming to Alex to sell some of the possessions of a physicist. Chase (Alex's assistant, and here, as in the last few books, the viewpoint character) doesn't even recognize the name, but Alex does: Christopher Robin disappeared forty-one years earlier, under circumstances that were never fully explained. He was also (and unusually for a physicist) interested in strange and marginal ideas: dark energy, new drive technology, parallel universes one could potentially cross into, and similar fringe concepts. The normal pattern of this series is that Alex will hear about some mystery, be unable to restrain his curiosity, and start poking around, usually turning up things that people would rather he didn't. It takes a while for that to happen here; instead, the story starts with Alex playing up popular interest in Robin's ideas in a rather mercenary attempt to increase the value of the estate. He stumbles into more mystery mostly by accident. Eventually he can't resist the allure of a revealed link between Christopher Robin and sightings of mysterious disappearing starships and the normal pattern kicks in, but he spends rather more of the book than normal being flippant and slightly unethical. I didn't much like the shift in tone. Alex is a lot harder to like this book, and not just for his business practices. His tone towards Chase also moves past the slightly superior smugness that's common to many books of this type (think Nero Wolfe) and well into condescending ass. Some of this may be intentional, as McDevitt uses this book to bring out a bit more of Alex and Chase's past and has some story reasons for making Alex less of a saint. But some of it feels accidental, or unnoticed, and it left a bad taste in my mouth. As with a lot of McDevitt, the actual mystery is slow, a bit scattered, and has a fair number of blind alleys. That property makes these books feel more like real investigations, but it works better when the characters are fully engaged in the investigation and are communicating a bit better than they are here. The plot also gets entangled in a subplot about a planet full of abandoned AIs, and while that was moderately interesting, it felt like an extended digression with dubious relevance to the main plot. McDevitt occasionally has trouble with plot focus, and I think I noticed more this time because the characters weren't as fun to spend time with. The end of Firebird was up to the usual standards of this series, albeit surprisingly traumatic. The rest of the book, though, felt markedly weaker. Alex decided to be obnoxious and play into his (previously mostly inaccurate) public perception as a money and glory hound. Chase seemed too flippant, uninterested, and a bit too much of a foil instead of a partner. And the plot felt like two different story ideas awkwardly smushed together. I think this was the weakest of the series to date. Hopefully the next book, which looks to be more of a direct sequel to this one, will improve. Followed by Coming Home. Rating: 6 out of 10

14 August 2015

Alberto Garc a: I/O limits for disk groups in QEMU 2.4

QEMU 2.4.0 has just been released, and among many other things it comes with some of the stuff I have been working on lately. In this blog post I am going to talk about disk I/O limits and the new feature to group several disks together. Disk I/O limits Disk I/O limits allow us to control the amount of I/O that a guest can perform. This is useful for example if we have several VMs in the same host and we want to reduce the impact they have on each other if the disk usage is very high. The I/O limits can be set using the QMP command block_set_io_throttle, or with the command line using the throttling.* options for the -drive parameter (in brackets in the examples below). Both the throughput and the number of I/O operations can be limited. For a more fine-grained control, the limits of each one of them can be set on read operations, write operations, or the combination of both: Example: 
-drive if=virtio,file=hd1.qcow2,throttling.bps-write=52428800,throttling.iops-total=6000
In addition to that, it is also possible to configure the maximum burst size, which defines a pool of I/O that the guest can perform without being limited: One additional parameter named iops_size allows us to deal with the case where big I/O operations can be used to bypass the limits we have set. In this case, if a particular I/O operation is bigger than iops_size then it is counted several times when it comes to calculating the I/O limits. So a 128KB request will be counted as 4 requests if iops_size is 32KB. Group throttling All of these parameters I ve just described operate on individual disk drives and have been available for a while. Since QEMU 2.4 however, it is also possible to have several drives share the same limits. This is configured using the new group parameter. The way it works is that each disk with I/O limits is member of a throttle group, and the limits apply to the combined I/O of all group members using a round-robin algorithm. The way to put several disks together is just to use the group parameter with all of them using the same group name. Once the group is set, there s no need to pass the parameter to block_set_io_throttle anymore unless we want to move the drive to a different group. Since the I/O limits apply to all group members, it is enough to use block_set_io_throttle in just one of them. Here s an example of how to set groups using the command line: 
-drive if=virtio,file=hd1.qcow2,throttling.iops-total=6000,throttling.group=foo
-drive if=virtio,file=hd2.qcow2,throttling.iops-total=6000,throttling.group=foo
-drive if=virtio,file=hd3.qcow2,throttling.iops-total=3000,throttling.group=bar
-drive if=virtio,file=hd4.qcow2,throttling.iops-total=6000,throttling.group=foo
-drive if=virtio,file=hd5.qcow2,throttling.iops-total=3000,throttling.group=bar
-drive if=virtio,file=hd6.qcow2,throttling.iops-total=5000
In this example, hd1, hd2 and hd4 are all members of a group named foo with a combined IOPS limit of 6000, and hd3 and hd5 are members of bar. hd6 is left alone (technically it is part of a 1-member group). Next steps I am currently working on providing more I/O statistics for disk drives, including latencies and average queue depth on a user-defined interval. The code is almost ready. Next week I will be in Seattle for the KVM Forum where I will hopefully be able to finish the remaining bits.
I will also attend LinuxCon North America. Igalia is sponsoring the event and we have a booth there. Come if you want to talk to us or see our latest demos with WebKit for Wayland. See you in Seattle!

Next.

Previous.